NATURAL LANGUAGE INFERENCE

Two-tiered Encoder-based Hallucination Detection for Retrieval-Augmented Generation in the Wild (2024.emnlp-industry)

Challenge:	Recent studies have focused on the application and evaluation of Large Language Models (LLMs) but LLMs are still prone to factual errors and inconsistencies in their explanations, offering limited control and interpretability for inference in complex domains.
Approach:	They propose an abductive-deductive framework that integrates Large Language Models with an external backward-chaining solver to refine step-wise natural language explanations.
Outcome:	The proposed framework improves explanations generated via in-context learning methods and Chain-of-Thought (CoT) on ethical NLI tasks while producing formal proofs describing and supporting models’ reasoning.

Copied to clipboard

Ilana Zimmerman, Jadin Tredup, Ethan Selfridge, Joseph Bradley

A Comparative Analysis of Unsupervised Language Adaptation Methods (D19-61)

Challenge:	Existing solutions for hallucination detection do not consider latency, train or evaluate on production data.
Approach:	They propose to use customer service conversation data to evaluate existing methods . they propose to train small encoder models on a new dataset to outperform existing methods.
Outcome:	The proposed model outperforms existing methods and highlights the value of combining small amounts of in-domain data with public datasets.

Copied to clipboard

Gil Rocha, Henrique Lopes Cardoso

Learning as Abduction: Trainable Natural Logic Theorem Prover for Natural Language Inference (2020.starsem-1)

Challenge:	Recent proposed approaches to perform unsupervised language adaptation lack annotated resources in less-resourced languages.
Approach:	They propose to use Adversarial Training, Sentence Encoder Alignment and Shared-Private Architecture to perform unsupervised language adaptation without using aligned sentences.
Outcome:	The proposed approaches are more suitable when the source and target language datasets contain other variations in content besides the language shift.

Copied to clipboard

Lasha Abzianidze

Jack the Reader – A Machine Reading Framework (P18-4)

Challenge:	a logic-based approach to Natural Language Inference is becoming less and less common . a new method uses semantic relations to abduct sentences from data .
Approach:	They propose a method to reverse a theorem-proving procedure to abduct semantic relations from data.
Outcome:	The proposed method improves the performance of the theorem prover on the SICK dataset by 1.4% while maintaining high precision (>94%)

Copied to clipboard

Dirk Weissenborn, Pasquale Minervini, Isabelle Augenstein, Johannes Welbl, Tim Rocktäschel, Matko Bošnjak, Jeff Mitchell, Thomas Demeester, Tim Dettmers, Pontus Stenetorp, Sebastian Riedel

Natural Language Interfaces

Challenge:	Many Machine Reading and Natural Language Understanding tasks require reading supporting text in order to answer questions.
Approach:	They propose a framework for Machine Reading that allows for quick prototyping by component reuse and evaluation of new models on existing datasets.
Outcome:	The proposed framework supports question answering, natural language inference and link prediction tasks.

Semantic Diversity in Dialogue with Natural Language Inference (2022.naacl-main)

Copied to clipboard

Katherine Stasaski, Marti Hearst

Dialogue Systems & Conversational Agents

Challenge:	Existing methods to measure diversity of chitchat model responses have been proposed to measure iteratively.
Approach:	They propose a metric which uses Natural Language Inference to measure the semantic diversity of a set of model responses for a conversation.
Outcome:	The proposed metric improves the diversity of a sampled set of responses using a new generation procedure called Diversity Threshold Generation.

NeuralLog: Natural Language Inference with Joint Neural and Logical Reasoning (2021.starsem-1)

Copied to clipboard

Zeming Chen, Qiyue Gao, Lawrence S. Moss

Logical Inference for Counting on Semi-structured Tables (2022.acl-srw)

Challenge:	Currently, symbolic and deep learning approaches to NLI are receiving less attention.
Approach:	They propose a symbolic-based inference framework that integrates symbolic reasoning and semantic formalism to solve NLI tasks.
Outcome:	The proposed framework improves accuracy on the NLI task and on the SICK and MED datasets.

Copied to clipboard

Tomoya Kurosawa, Hitomi Yanaka

Structured Data in NLP

Challenge:	Natural Language Inference (NLI) tasks require numerical understanding to perform a numerical type of inference, such as counting.
Approach:	They propose a logical inference system for reasoning between semi-structured tables and texts that uses logical representations as meaning representations and model checking to handle a numerical type of inference.
Outcome:	The proposed system can perform inference with numerical comparatives with tables and texts in English.

Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models (2023.acl-srw)

Copied to clipboard

Tomoki Sugimoto, Yasumasa Onoe, Hitomi Yanaka

XplaiNLI: Explainable Natural Language Inference through Visual Analytics (2020.coling-demos)

Challenge:	Existing datasets and analyses focusing on English do not address the need for resources in other languages.
Approach:	They propose a Japanese NLI benchmark focused on temporal inference . they use a set of temporal patterns to generate diverse examples .
Outcome:	The proposed model can perform fine-grained analysis in Japanese and English.

Copied to clipboard

Aikaterini-Lida Kalouli, Rita Sevastjanova, Valeria de Paiva, Richard Crouch, Mennatallah El-Assady

Visual Data in NLP

Challenge:	Recent research has revealed some heuristics and biases of natural language inference models.
Approach:	They propose an interactive visualization interface that computes Natural Language Inference with different methods and provides explanations for the decisions made by each approach.
Outcome:	The proposed interface computes Natural Language Inference (NLI) with three different approaches and provides explanations for the decisions made by each approach.

Compositional Semantics and Inference System for Temporal Order based on Japanese CCG (2022.acl-srw)

Copied to clipboard

Tomoki Sugimoto, Hitomi Yanaka

Exploring Factual Entailment with NLI: A News Media Study (2024.starsem-1)

Challenge:	a system for temporal order in Japanese has not been developed for linguistic inference involving temporal expressions.
Approach:	They propose a Japanese NLI system that considers temporal order in Japanese . they use axioms for temporal relations and automated theorem provers to perform inference involving temporal orders.
Outcome:	The proposed system outperforms logic-based systems and current deep learning models on Japanese datasets.

Copied to clipboard

Guy Mor-Lan, Effi Levi

Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning (2021.findings-acl)

Challenge:	Recent studies have focused on the relationship between factuality and Natural Language Inference (NLI).
Approach:	They propose a novel annotation scheme that models factual rather than textual entailment and use it to annotate a dataset of naturally occurring sentences from news articles.
Outcome:	The proposed annotation scheme can be used to model factual relationships on a dataset of naturally occurring sentences from news articles.

Copied to clipboard

Benjamin Minixhofer, Milan Gritta, Ignacio Iacobacci

Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Models (2024.eacl-long)

Challenge:	Recent advances in transfer learning have brought significant improvements to many natural language processing tasks.
Approach:	They propose a method of fitting a GBDT head on the features computed during finetuning to increase performance without additional computation by the neural network.
Outcome:	The proposed method improves on several NLI datasets using a strong baseline model (RoBERTa-large) with MNLI pretraining.

Copied to clipboard

Erik Arakelyan, Zhaoqi Liu, Isabelle Augenstein

Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters (2022.findings-emnlp)

Challenge:	Recent studies have indicated that NLI models have an understanding of lexical and compositional semantics.
Approach:	They propose a framework to assess the extent of semantic sensitivity in NLI models . they use adversarially generated examples with minor semantics-preserving surface-form variations .
Outcome:	The proposed framework shows that NLI models struggle with minor variations requiring knowledge of compositional semantics .

Copied to clipboard

Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Donald Metzler

Information Extraction & Text Mining

Challenge:	Recent advances in modeling and datasets demonstrate promising performance for NLI.
Approach:	They explore the direct zero-shot applicability of NLI models to real applications . they analyze the robustness of models to longer and out-of-domain inputs .
Outcome:	The proposed models are robust to longer and out-of-domain inputs and can perform on full documents.

Adversarial Text Normalization (2022.naacl-industry)

Copied to clipboard

Joanna Bitton, Maya Pavlova, Ivan Evtimov

Syntactic Text Processing

Challenge:	Text-based adversarial attacks are becoming more commonplace and accessible to general internet users.
Approach:	They propose a method that restores baseline performance on attacked content with low computational overhead.
Outcome:	The proposed method restores baseline performance on attacked content with low computational overhead.

Logical Inferences with Comparatives and Generalized Quantifiers (2020.acl-srw)

Copied to clipboard

Izumi Haruta, Koji Mineshima, Daisuke Bekki

Deep Neural Representations for Multiword Expressions Detection (2022.acl-srw)

Challenge:	Comparative constructions pose a challenge in Natural Language Inference (NLI) Comparatives interact with quantifiers, numerals, and lexical antonyms, but a logical inference system for comparatives has not been developed for the task.
Approach:	They propose a compositional semantics system that maps comparative constructions to semantic representations via combinatory categorial grammar parsers and integrates it with an automated theorem proving system.
Outcome:	The proposed system outperforms previous logic-based systems and deep learning models on three NLI datasets.

Copied to clipboard

Kamil Kanclerz, Maciej Piasecki

Representation Learning

Challenge:	Existing methods for multiword expression detection are based on sequence labeling and statistical measures.
Approach:	They propose a weakly supervised method for multiword expressions extraction . they use a lexicon of English multiword lexical units as a reference knowledge base .
Outcome:	The proposed method can be easily applied to other languages.

Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data (2023.acl-short)

Copied to clipboard

Ujan Deb, Ridayesh Parab, Preethi Jyothi

A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference (2024.lrec-main)

Challenge:	Zero-shot cross-lingual transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter within a source language.
Approach:	They propose to use unlabeled text to enhance zero-shot transfer by pairing language adapters with task adapters in a target language.
Outcome:	The proposed framework improves on three cross-lingual tasks with up to 11% relative improvement in Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI).

Copied to clipboard

Mokanarangan Thayaparan, Marco Valentino, André Freitas

Zero-Shot-BERT-Adapters: a Zero-Shot Pipeline for Unknown Intent Detection (2023.findings-emnlp)

Challenge:	Existing ILP frameworks are non-differentiable and cannot be integrated as part of a broader deep learning architecture.
Approach:	They propose a neuro-symbolic architecture for explanation-based NLI based on DBCS.
Outcome:	The proposed approach achieves superior performance when compared to existing solvers and black-box solver.

Copied to clipboard

Daniele Comi, Dimitrios Christofidellis, Pier Piazza, Matteo Manica

MENLI: Robust Evaluation Metrics from Natural Language Inference (2023.tacl-1)

Challenge:	Intent discovery remains a crucial task in natural language processing . identifying novel, unseen intents remains one of the biggest challenges in this field .
Approach:	They propose a multi-language approach to intent discovery using Adapters and a Transformer architecture.
Outcome:	The proposed pipeline outperforms baselines in two zero-shot settings for intent classification and unseen intent discovery.

Copied to clipboard

Yanran Chen, Steffen Eger

Leveraging Moment Injection for Enhanced Semi-supervised Natural Language Inference with Large Language Models (2025.naacl-short)

Challenge:	Recent proposed BERT-based evaluation metrics for text generation are vulnerable to adversarial attacks, e.g., relating to information correctness.
Approach:	They propose to use BERT-based evaluation metrics for text generation to evaluate text for semantic similarity but are vulnerable to adversarial attacks using Natural Language Inference.
Outcome:	The proposed metrics outperform existing summarization metrics but perform below SOTA MT metrics on standard benchmarks.

Copied to clipboard

Seo Yeon Park

Frame-Semantic Knowledge Injection for Event-Level Inference in LLMs (2026.acl-short)

Challenge:	Existing studies have used class-specific fine-tuned large language models to generate hypotheses and assign pseudo-labels but discarded many LLM-constructed samples to ensure the quality.
Approach:	They propose to leverage LLM-constructed samples by injecting the moments of labeled samples during training to properly adjust the level of noise.
Outcome:	The proposed method outperforms strong baselines on multiple NLI datasets in low-resource settings.

Copied to clipboard

Shahid Iqbal Rai, Danilo Croce, Roberto Basili

PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise (2026.acl-long)

Challenge:	Large language models (LLMs) are fluent but often brittle when interpretation depends on external information.
Approach:	They propose a framework that injects frame-semantic knowledge into Large Language Models via LoRA.
Outcome:	The proposed framework can generalize beyond surface cues in large language models.

Copied to clipboard

Sapir Harary, Eran Hirsch, Aviv Slobodkin, David Wan, Mohit Bansal, Ido Dagan

Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness (2020.emnlp-main)

Challenge:	Large Language Models (LLMs) are prone to factually inconsistent statements, known as hallucinations.
Approach:	They propose to train a specialized model that detects inconsistencies over text prefixes to improve generation faithfulness by 5-14 F1 points.
Outcome:	The proposed model outperforms baseline models by 5-14 F1 points in prefix-level entailment.

Copied to clipboard

Hyunwoo Kim, Byeongchang Kim, Gunhee Kim

Dialogue Systems & Conversational Agents

Challenge:	Existing models for improving consistency often train with additional NLI labels or attach trained extra modules to the generative agent.
Approach:	They propose to encode personas into dialogue embeddings and a persona-conditioned dialogue dataset to improve persona consistency.
Outcome:	The proposed approach can enforce dialogue agents to refrain from contradictions and improve consistency of existing models.

Two-Step Classification using Recasted Data for Low Resource Settings (2020.aacl-main)

Copied to clipboard

Shagun Uppal, Vivek Gupta, Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, Amanda Stent

Information Extraction & Text Mining

Challenge:	Existing studies on NLP models focus on high resource languages like English, but there are only two datasets for Hindi.
Approach:	They propose a novel two-step classification method which uses textual-entailment predictions for classification task.
Outcome:	The proposed method improves classification performance by using a joint-objective for classification and textual entailment.

Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs (2025.findings-naacl)

Copied to clipboard

Itai Mondshine, Tzuf Paz-Argaman, Reut Tsarfaty

Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)

Challenge:	Current LLMs are primarily trained on English data but also include data from other languages.
Approach:	They propose to use a pre-translation strategy to translate a task prompt into English before inference . they use 'a modular entity' that could be translated into four different languages .
Outcome:	The proposed strategies are based on a set of pre-trained data across 35 languages covering both low and high-resource languages.

Copied to clipboard

Hitomi Yanaka, Koji Mineshima

To be or not to be an Integer? Encoding Variables for Mathematical Text (2022.findings-acl)

Challenge:	Despite growing interest in linguistic universals, most NLI/STS studies focus on English.
Approach:	They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK.
Outcome:	The proposed datasets show that pre-trained language models are insensitive to word order and case particles.

Copied to clipboard

Deborah Ferreira, Mokanarangan Thayaparan, Marco Valentino, Julia Rozanova, Andre Freitas

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference (P19-1)

Challenge:	a number of natural language inference models are limited in interpreting mathematical knowledge written in Natural Language . a variable's meaning is determined exclusively by its defining type, i.e., its context .
Approach:	They propose a method that can create context-based representations for variables . they propose 'variable slot' approach which can be used to model variables based on their meaning .
Outcome:	The proposed model can be used to represent variables in natural language . it can be applied to a task of variable typing and create context-based representations for variables .

Copied to clipboard

Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, Alexander Rush

Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference (P18-1)

Challenge:	Natural Language Inference (NLI) datasets often contain hypothesis-only biases . authors propose probabilistic methods to build models that are more robust to such artifacts - a new study shows .
Approach:	They propose probabilistic methods to build models that are more robust to biases . authors train on datasets containing biase .
Outcome:	The proposed methods can make NLI models more robust to dataset-specific artifacts . the methods transfer better than a baseline architecture in 9 out of 12 NLI datasets compared with baseline architectures based on the proposed methods .

Copied to clipboard

Boyuan Pan, Yazheng Yang, Zhou Zhao, Yueting Zhuang, Deng Cai, Xiaofei He

Generating Token-Level Explanations for Natural Language Inference (N19-1)

Challenge:	Existing approaches to natural language inference focus on interaction architectures of sentences . but, we propose to transfer knowledge from discourse markers to augment the model .
Approach:	They propose to transfer knowledge from discourse markers to augment the quality of the NLI model.
Outcome:	The proposed method achieves state-of-the-art performance on large-scale datasets.

Copied to clipboard

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal

Information Retrieval

Challenge:	Existing methods to generate token-level explanations for NLI on single sentences have not been tested.
Approach:	They propose to generate token-level explanations for NLI without explicitly annotating training data.
Outcome:	The proposed approach is faster and more accurate than the black-box methods.

A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference (2022.emnlp-main)

Copied to clipboard

Kerem Zaman, Yonatan Belinkov

Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024.eacl-long)

Challenge:	Existing evaluations of attribution methods focus on the English language . plausibility and faithfulness are two main criteria for plausible and faithful attributions .
Approach:	They propose a cross-lingual strategy to measure faithfulness based on word alignments.
Outcome:	The proposed approach eliminates drawbacks of erasure-based evaluations and provides a multilingual dataset with highlights to support future studies.

Copied to clipboard

Huajian Zhang, Yumo Xu, Laura Perez-Beltrachini

Summarization

Challenge:	Existing approaches to evaluate summary faithfulness are sub-optimal due to the granularity level considered for premises and hypotheses.
Approach:	They propose a novel approach that uses a variable premise size and simplifies summary sentences into shorter hypotheses.
Outcome:	The proposed model performs better on diverse summarisation tasks than existing models.

Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021.naacl-main)

Copied to clipboard

Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula

Fast and Accurate Factual Inconsistency Detection Over Long Documents (2023.emnlp-main)

Challenge:	Large-scale, open Natural Language Inference datasets have catalyzed the development of NLI models that exhibit close to human-level performance, but the use of these models for other downstream NLP tasks has met with limited success.
Approach:	They use multiple-choice reading comprehension and checking factual correctness of textual summarization tasks to investigate potential reasons for this . authors leverage abundance of data from reading comprehension datasets into longer-premise NLI datasets to improve their models .
Outcome:	The proposed models outperform models trained on converted datasets due to the difference in premise lengths.

Copied to clipboard

Barrett Lattimer, Patrick CHen, Xinyuan Zhang, Yi Yang

Chunking

Challenge:	Generative AI models exhibit remarkable potential, however, hallucinations across various tasks present a significant challenge, particularly for longer inputs.
Approach:	They propose a task-agnostic model that uses large text chunks to condition over long texts and employ a novel algorithm to explain its decisions through relevant source sentence retrieval.
Outcome:	The proposed model outperforms existing methods on benchmarks and a new long-form dialogue dataset and surpasses competitive systems in efficiency and model explanation evaluations.

Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generation (2024.findings-naacl)

Copied to clipboard

Yifu Qiu, Varun Embar, Shay Cohen, Benjamin Han

Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims (2022.naacl-main)

Challenge:	Knowledge-to-text generators often struggle to faithfully generate descriptions for input facts . we propose a decoding-only method to reduce hallucinations .
Approach:	They propose a decoding-only method to generate accurate descriptions for input facts . they use a Natural Language Inference model as the model and replace it with a task-specific HVM .
Outcome:	The proposed method improves faithfulness with minimal impact on quality and in/out-of-distribution evaluations.

Copied to clipboard

Miguel Arana-Catania, Elena Kochkina, Arkaitz Zubiaga, Maria Liakata, Robert Procter, Yulan He

Embracing Ambiguity: Shifting the Training Target of NLI Models (2021.acl-short)

Challenge:	Existing datasets focus on a single medium, information domain or specific application . authors propose novel methods for automated veracity assessment based on Natural Language Inference .
Approach:	They propose to build a PANACEA dataset that combines different data sources with different foci to ensure a unique set of claims.
Outcome:	The proposed methods are competitive with SOTA methods and provide a detailed discussion.

Copied to clipboard

Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, Akiko Aizawa

Challenge:	Previously, it was common to disregard ambiguity as noise or as a sign of poor quality data.
Approach:	They propose to train on the estimated label distribution of annotators in a NLI task . they use a learning loss based on this ambiguity distribution instead of gold-labels .
Outcome:	The proposed training method reduces divergence scores on a trial dataset . the study shows that targeting the ambiguity distribution can improve performance .

Assessing Monotonicity Reasoning in Dutch through Natural Language Inference (2023.findings-eacl)

Copied to clipboard

Gijs Wijnholds

Cooperative Learning of Disjoint Syntax and Semantics (N19-1)

Challenge:	a novel dataset for natural language inference (NLI) is used to study monotonicity reasoning in Dutch.
Approach:	They investigate monotonicity reasoning in Dutch using a novel dataset . they find that models struggle with downward entailing contexts .
Outcome:	The proposed dataset shows that models struggle with downward entailing contexts, and argue that this is due to a poor understanding of negation.

Copied to clipboard

Serhii Havrylov, Germán Kruszewski, Armand Joulin

Syntactic Text Processing

Challenge:	Existing models that learn to jointly infer an expression’s syntactic structure and its semantics fail to learn the correct parsing strategy on mathematical expressions generated from a simple context-free grammar.
Approach:	They propose a recursive model that learns to jointly infer an expression’s syntactic structure and its semantics without requiring a formal supervision.
Outcome:	The proposed model performs competitively on several natural language tasks, such as Natural Language Inference and Sentiment Analysis.

A synthetic data approach for domain generalization of NLI models (2024.acl-long)

Copied to clipboard

Mohammad Javad Hosseini, Andrey Petrov, Alex Fabrikant, Annie Louis

Challenge:	Natural Language Inference (NLI) datasets are important benchmark tasks for LLMs . however, their realistic performance on out-of-distribution/domain data is less well-understood . a T5-small model trained with our data improves around 7% on average compared to the best alternative dataset .
Approach:	They propose a new approach for generating NLI data in diverse domains and lengths . they show that models trained on this data have the best generalization to completely new downstream test settings .
Outcome:	The proposed model can be trained on datasets with high-quality examples with meaningful premises and high accuracy.

Capture Human Disagreement Distributions by Calibrated Networks for Natural Language Inference (2022.findings-acl)

Copied to clipboard

Yuxia Wang, Minghan Wang, Yimeng Chen, Shimin Tao, Jiaxin Guo, Chang Su, Min Zhang, Hao Yang

SICK-NL: A Dataset for Dutch Natural Language Inference (2021.eacl-main)

Challenge:	Previously, it's common to disregard it as noise or as a sign of poor-quality data, as their annotations are heavily based on personal experience and opinions.
Approach:	They propose to capture the human disagreement distribution from the perspective of model calibration.
Outcome:	The proposed model can achieve competitive performance when well-calibrated, on divergence scores between predictive probability and the true human opinion distribution, and the accuracy.

Copied to clipboard

Gijs Wijnholds, Michael Moortgat

TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models (2023.emnlp-main)

Challenge:	Having a parallel dataset for Natural Language Inference in Dutch is problematic for some NLP systems.
Approach:	They propose to translate a SICK dataset from English into Dutch to compare models for both languages.
Outcome:	The proposed dataset compares models on English and Dutch on two tasks.

Copied to clipboard

Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, Idan Szpektor

Less for More: Enhanced Feedback-aligned Mixed LLMs for Molecule Caption Generation and Fine-Grained NLI Evaluation (2025.acl-long)

Challenge:	Existing methods for evaluating factual consistency are limited in their effectiveness.
Approach:	They propose a method for generating synthetic data by annotating diverse model-generated summaries using a LLM.
Outcome:	The proposed method outperforms state-of-the-art models and the LLM teacher on TRUE benchmarks.

Copied to clipboard

Dimitris Gkoumas, Maria Liakata

Text Generation

Challenge:	Recent trends have led to the use of multimodal models to learn molecular and linguistic representations, either in separate but coordinated spaces or in a common space.
Approach:	They propose a novel atomic-level evaluation method leveraging off-the-shelf Natural Language Inference (NLI) models for use in the unseen chemical domain.
Outcome:	The proposed method surpasses state-of-the-art models in the unseen chemical domain while relying on a granularity-based evaluation method.

Uncertainty-Aware Cross-Lingual Transfer with Pseudo Partial Labels (2022.findings-naacl)

Copied to clipboard

Shuo Lei, Xuchao Zhang, Jianfeng He, Fanglan Chen, Chang-Tien Lu

Unsupervised Natural Language Inference Using PHL Triplet Generation (2022.findings-acl)

Challenge:	Existing methods to train pre-trained language models for zero-shot cross-lingual tasks are noisy and lack confidence.
Approach:	They propose an uncertainty-aware cross-lingual transfer framework with pseudo-partial-label to maximize the utilization of unlabeled data by reducing noise.
Outcome:	The proposed framework outperforms baselines on named entity recognition and natural language inference tasks on 40 languages.

Copied to clipboard

Neeraj Varshney, Pratyay Banerjee, Tejas Gokhale, Chitta Baral

Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving (2024.emnlp-main)

Challenge:	In some cases, training samples may not be available or collecting them could be time-consuming and resource-intensive.
Approach:	They propose a procedural approach that leverages sentence transformations to collect PHL triplets for training NLI models.
Outcome:	The proposed model outperforms existing models on several NLI benchmarks with a set of sentence transformations.

Copied to clipboard

Xin Quan, Marco Valentino, Louise Dennis, Andre Freitas

Evaluating the Robustness of Discrete Prompts (2023.eacl-main)

Challenge:	Existing methods for assessing the validity of explanations for NLI are time-consuming and prone to logical errors.
Approach:	They propose a framework that integrates Large Language Models and Theorem Provers to verify and refine natural language explanations through crowd-sourcing . they propose to use TPs to generate and formalise explanatory sentences and suggest potential inference strategies for NLI.
Outcome:	The proposed framework generates and formalises explanatory sentences and suggests potential inference strategies for NLI.

Copied to clipboard

Yoichi Ishibashi, Danushka Bollegala, Katsuhito Sudoh, Satoshi Nakamura

How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics (2024.emnlp-main)

Challenge:	Existing methods that generate discrete prompts from a small set of training instances have reported superior performance, but manual writing prompts that generalize well is challenging due to several reasons.
Approach:	They propose to use discrete prompts to learn lexical constructs that would not be encountered in manually-written prompts.
Outcome:	The proposed method is robust against perturbations to NLI inputs but sensitive to other types of perturbations such as shuffling and deletion of prompt tokens.

Copied to clipboard

Adrian Cosma, Stefan Ruseti, Mihai Dascalu, Cornelia Caragea

Challenge:	Popular datasets suffer from systematic spurious correlations that artificially inflate actual model performance.
Approach:	They propose a method for the automated creation of a challenging test set without relying on manual construction of artificial and unrealistic examples.
Outcome:	The proposed method reduces spurious correlations and improves model performance . examples labeled as having the highest difficulty show markedly decreased performance compared to the full dataset .

Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness (N18-1)

Copied to clipboard

Ivan Sanchez, Jeff Mitchell, Sebastian Riedel

Sources of Hallucination by Large Language Models on Inference Tasks (2023.findings-emnlp)

Challenge:	Currently, state-of-the-art models achieve impressive test set performance in the form of accuracy scores.
Approach:	They examine the models' robustness to semantically-valid alterations to the input data by identifying three factors and comparing their impact on three SNLI models.
Outcome:	The proposed models show that they can generalise to new in-domain instances while also showing that they suffer from insensitivity to small but semantically significant alterations.

Copied to clipboard

Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Hosseini, Mark Johnson, Mark Steedman

Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference (D18-1)

Challenge:	Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI)
Approach:	They propose to use LLMs to probe their behavior using controlled experiments.
Outcome:	The proposed models perform significantly worse on NLI test samples which do not conform to these biases than those which do.

Copied to clipboard

Yi Tay, Anh Tuan Luu, Siu Cheung Hui

Language Model Analysis for Ontology Subsumption Inference (2023.findings-acl)

Challenge:	Using a new architecture, alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning.
Approach:	They propose a new architecture where alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning.
Outcome:	The proposed architecture achieves competitive performance on three popular benchmarks, SNLI, MultiNLI and SciTail, while maintaining lightweight parameter size.

Copied to clipboard

Yuan He, Jiaoyan Chen, Ernesto Jimenez-Ruiz, Hang Dong, Ian Horrocks

EENLP: Cross-lingual Eastern European NLP Index (2022.lrec-1)

Challenge:	Existing studies focus on simple, triple-based, relational KBs but omit more sophisticated, logic-based conceptualised KB.
Approach:	They propose to use ontology subsumption axioms to probe LMs' knowledge of ontologies by probing datasets from atomic and complex concepts.
Outcome:	The proposed methods encode less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference but can improve on SI significantly when a small number of samples are given.

Copied to clipboard

Alexey Tikhonov, Alex Malkhasov, Andrey Manoshin, George-Andrei Dima, Réka Cserháti, Md.Sadek Hossain Asif, Matt Sárdi

Sentiment Analysis

Challenge:	Existing NLP resources for Eastern European languages are sparse.
Approach:	They propose to use existing Eastern European language resources to build cross-lingual datasets for five different semantic tasks to support commonsense reasoning.
Outcome:	The proposed model trains on 104 languages and shows impressive results on text analysis tasks.

Deep Natural Language Feature Learning for Interpretable Prediction (2023.emnlp-main)

Copied to clipboard

Felipe Urrutia, Cristian Calderon, Valentin Barriere

Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (2020.lrec-1)

Challenge:	Using a small transformer language model, we can break down a complex task into a set of intermediary easier sub-tasks.
Approach:	They propose a method to break down a main task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task.
Outcome:	The proposed method breaks down a complex task into a set of easier sub-tasks, which are formulated in natural language as binary questions related to the final target task.

Copied to clipboard

Carlos Aspillaga, Andrés Carvallo, Vladimir Araujo

XNLIeu: a dataset for cross-lingual NLI in Basque (2024.naacl-long)

Challenge:	Existing models are weak and take advantage of failures and errors in datasets to improve performance.
Approach:	They evaluate three Transformer-based models in Natural Language Inference and Question Answering tasks to see if they are more robust or have the same flaws as their predecessors.
Outcome:	The proposed models outperform recurrent neural network models to stress tests on both NLI and QA tasks.

Copied to clipboard

Maite Heredia, Julen Etxaniz, Muitze Zulaika, Xabier Saralegi, Jeremy Barnes, Aitor Soroa

Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models (2022.emnlp-main)

Challenge:	XNLI is a popular benchmark used to evaluate cross-lingual Natural Language Understanding (NLU) in languages such as English, Basque and other low-resource languages.
Approach:	They expand XNLI to include Basque, a low-resource language that can benefit from transfer-learning approaches.
Outcome:	The proposed dataset includes Basque, a low-resource language that can benefit from transfer-learning approaches.

Copied to clipboard

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Marek Rei

Dually Self-Improved Counterfactual Data Augmentation Using Large Language Model (2025.acl-long)

Challenge:	Current models learn from annotation artefacts and dataset biases, but it is unclear to what extent they are learning the task of NLI.
Approach:	They propose a logical reasoning framework that allows models to learn from annotation artefacts and dataset biases.
Outcome:	The proposed model outperforms humans on in-distribution test sets without using span labels . the model is more robust in a reduced data setting, and out-of-disturbance performance is improved .

Copied to clipboard

Luhao Zhang, Xinyu Zhang, Linmei Hu, Dandan Song, Liqiang Nie

Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)

Challenge:	Existing approaches to generate counterfactual data augmentation are limited due to imbalance and biases in real-world training data.
Approach:	They propose a self-improved method for generating high-quality counterfacts using large language models.
Outcome:	The proposed method generates high-quality counterfacts on the natural language inference task using lightweight and task-specific LLMs.

Copied to clipboard

Joe Stacey, Lisa Alazraki, Aran Ubhi, Beyza Ermis, Aaron Mueller, Marek Rei

Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)

Challenge:	Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models.
Approach:	They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets.
Outcome:	The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data.

Copied to clipboard

Abhilasha Ravichander, Yonatan Belinkov, Eduard Hovy

Realistic Data Augmentation Framework for Enhancing Tabular Reasoning (2022.findings-emnlp)

Challenge:	Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate.
Approach:	They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself.
Outcome:	The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks.

Copied to clipboard

Dibyakanti Kumar, Vivek Gupta, Soumya Sharma, Shuo Zhang

Multimodality

Challenge:	Existing approaches to constructing training data for Natural Language Inference (NLI) tasks are expensive and time consuming.
Approach:	They propose a semi-automated framework for data augmentation for tabular inference . framework generates hypothesis templates transferable to similar tables . authors say framework could generate human-like tabular examples .
Outcome:	The proposed framework generates human-like tabular inference examples . it is based on human-written constraints and premise paraphrasing .

GLUECoS: An Evaluation Benchmark for Code-Switched NLP (2020.acl-main)

Copied to clipboard

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

A Deep Transfer Learning Method for Cross-Lingual Natural Language Inference (2022.lrec-1)

Challenge:	Recent studies show multilingual contextual embedding models perform better on cross-lingual and multilingual tasks.
Approach:	They propose to evaluate multilingual contextual embedding models on multilingual data . they use language identification from text, POS tagging, Named Entity Recognition and Question Answering .
Outcome:	The proposed benchmark evaluates models on language identification from text, POS tagging, Named Entity Recognition, Question Answering and a new task for code-switching, Natural Language Inference.

Copied to clipboard

Dibyanayan Bandyopadhyay, Arkadipta De, Baban Gain, Tanik Saikh, Asif Ekbal

An Imitation Learning Approach to Unsupervised Parsing (P19-1)

Challenge:	Natural Language Inference (NLI) is a crucial task in AI and natural language processing.
Approach:	They propose an effective transfer learning approach for cross-lingual NLI . they perform experiments on English-Hindi language pairs in cross-linguistic setting .
Outcome:	The proposed model improves the baseline model by 10% over the state-of-the-art model.

Copied to clipboard

Bowen Li, Lili Mou, Frank Keller

InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples (2022.coling-1)

Challenge:	Unsupervised parsing is a form of reinforcement learning that improves syntactic structures but lacks interpretability due to its lack of ad hoc heuristics.
Approach:	They propose an unsupervised approach that transfers syntactic knowledge to a Tree-LSTM model with discrete parsing actions.
Outcome:	The proposed model outperforms existing models on the All Natural Language Inference dataset and achieves a new state of the art in terms of parsing F-score.

Copied to clipboard

Venelin Kovatchev, Mariona Taulé

Learning to Infer from Unlabeled Data: A Semi-supervised Learning Approach for Robust Natural Language Inference (2022.findings-emnlp)

Challenge:	InferES is an original corpus for Natural Language Inference (NLI) in European Spanish .
Approach:	They propose to implement and analyze a corpus-creating strategy utilizing expert linguists and crowd workers to provide high-quality data and facilitate the systematic evaluation of automated systems.
Outcome:	The proposed model obtains 72.8% accuracy and performs moderately well on negation-based adversarial examples.

Copied to clipboard

Mobashir Sadat, Cornelia Caragea

SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)

Challenge:	Semi-supervised learning (SSL) is a popular technique for reducing the reliance on human annotations for NLI tasks.
Approach:	They propose a way to incorporate unlabeled data into semi-supervised learning (SSL) using a conditional language model, they propose to generate hypotheses for unlabed sentences .
Outcome:	The proposed framework significantly improves the performance of four NLI datasets in low-resource settings.

Copied to clipboard

Magdalena Wysocka, Danilo Carvalho, Oskar Wysocki, Marco Valentino, Andre Freitas

Using Natural Language Relations between Answer Choices for Machine Comprehension (N19-1)

Challenge:	Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications.
Approach:	They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence.
Outcome:	The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways.

Copied to clipboard

Rajkumar Pujari, Dan Goldwasser

Natural Language Interfaces

Challenge:	Current approaches to the reading comprehension task quantify the relationship between each question and answer choice independently and pick the highest scoring option.
Approach:	They propose a method to leverage natural language relations between answer choices to improve machine comprehension.
Outcome:	The proposed model improves the performance of a reading comprehension task by leveraging natural language relations between answer choices.

LCHAIM - Investigating Long Context Reasoning in Hebrew (2025.findings-acl)

Copied to clipboard

Ehud Malul, Oriel Perets, Ziv Mor, Yigal Kassel, Elior Sulem

Alignment Rationale for Natural Language Inference (2021.acl-long)

Challenge:	a dataset designed to evaluate long premises and complex reasoning models in Hebrew is being used in research.
Approach:	They introduce a dataset to evaluate Hebrew NLI models on long premises and complex reasoning tasks.
Outcome:	The proposed dataset evaluates models on tasks involving long premises and complex reasoning.

Copied to clipboard

Zhongtao Jiang, Yuanzhe Zhang, Zhao Yang, Jun Zhao, Kang Liu

Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference (2021.findings-emnlp)

Challenge:	Existing explanation methods pick prominent features, but alignments between words or phrases are more enlightening clues to explain the model.
Approach:	They propose a method to generate alignment rationale explanations for co-attention based models in NLI by feature selection.
Outcome:	The proposed method is more faithful and human-readable compared with existing methods.

Copied to clipboard

Canming Huang, Weinan He, Yongmei Liu

Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (2023.findings-emnlp)

Challenge:	Recent methods based on pre-trained language models have shown strong supervised performance on commonsense reasoning.
Approach:	They propose to use a common framework to solve commonsense reasoning tasks using a dataset from NLI.
Outcome:	The proposed method achieves state-of-the-art unsupervised performance on two commonsense reasoning tasks.

Copied to clipboard

Daichi Haraguchi, Kiyoaki Shirai, Naoya Inoue, Natthawut Kertkeidkachorn

ECON: On the Detection and Resolution of Evidence Conflicts (2024.emnlp-main)

Challenge:	Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model.
Approach:	They propose a method to quantify the severity of shortcut reasoning by leveraging out-of-distribution data.
Outcome:	The proposed method quantifies the severity of the discovered shortcut reasoning using out-of-distribution data.

Copied to clipboard

Cheng Jiayang, Chunkit Chan, Qianqian Zhuang, Lin Qiu, Tianhang Zhang, Tengxiao Liu, Yangqiu Song, Yue Zhang, Pengfei Liu, Zheng Zhang

Hy-NLI: a Hybrid system for Natural Language Inference (2020.coling-main)

Challenge:	Recent studies have shown that AI generated content is more likely to dominate search results, making it difficult to detect when compared to human-produced content.
Approach:	They propose a method for generating diverse, validated evidence conflicts to simulate real-world misinformation scenarios.
Outcome:	The proposed method enables the detection of conflicting information in real-world scenarios and shows that weaker models struggle with similar answer conflicts while stronger models show robust performance.

Copied to clipboard

Aikaterini-Lida Kalouli, Richard Crouch, Valeria de Paiva

Continual Few-Shot Learning for Text Classification (2021.emnlp-main)

Challenge:	Recent work has revealed the generalization difficulties of deep models . however, such models fail to perform well on adversarial datasets with challenging linguistic phenomena.
Approach:	They propose a hybrid system that learns to identify an NLI pair as linguistically challenging or not . their system uses symbolic or deep learning components to make the final inference decision .
Outcome:	The proposed system outperforms existing models on adversarial datasets and on mainstream datasets.

Copied to clipboard

Ramakanth Pasunuru, Veselin Stoyanov, Mohit Bansal

Low-Resource NLP

Challenge:	a large number of end-to-end systems are needed for many tasks in natural language processing.
Approach:	They propose a continual few-shot learning task where a system is asked to correct mistakes with a few training examples.
Outcome:	The proposed task compares two NLI and one sentiment analysis datasets with baselines from diverse paradigms.

Evaluating Numeracy of Language Models as a Natural Language Inference Task (2025.findings-naacl)

Copied to clipboard

Rahmad Mahendra, Damiano Spina, Lawrence Cavedon, Karin Verspoor

Pairwise Supervised Contrastive Learning of Sentence Representations (2021.emnlp-main)

Challenge:	Recent advances in large language models (LLMs) have enhanced their capabilities to solve mathematical problems, but other aspects of numeracy remain underexplored.
Approach:	They propose to frame numeracy as a Natural Language Inference task to assess the models’ ability to understand both numbers and language contexts.
Outcome:	The proposed model outperforms smaller models in arithmetic tasks, indicating that mathematical reasoning cannot be generalized to other numeracy skills such as number comparison and normalization.

Copied to clipboard

Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, Bing Xiang

Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning (2022.coling-1)

Challenge:	Recent efforts to improve sentence representation learning have a common weakness . siamese or triplet loss only learns from individual sentence pairs or tripletes .
Approach:	They propose a discrimination-based approach to bridge entailment and contradiction understanding with categorical concept encoding.
Outcome:	The proposed method outperforms the state-of-the-art method on downstream tasks . it improves 10%–13% on clustering tasks and 5%–6% on STS tasks compared with the previous method .

Copied to clipboard

Xuantao Lu, Jingping Liu, Zhouhong Gu, Hanwen Tong, Chenhao Xie, Junyang Huang, Yanghua Xiao, Wenguang Wang

Challenge:	Existing methods to parse natural language into structured logical expressions have limitations due to paucity of labeled data.
Approach:	They propose a scoring model to automatically learn a model-based reward . they also propose introducing a Chinese-PL/FOL dataset to compensate for paucity of labeled data .
Outcome:	The proposed model outperforms competitors on several datasets.

S2LPP: Small-to-Large Prompt Prediction across LLMs (2025.findings-emnlp)

Copied to clipboard

Liang Cheng, Tianyi Li, Zhaowei Wang, Mark Steedman

Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)

Challenge:	a small model can be used to select effective prompt templates for a larger model.
Approach:	They propose a method to use a smaller model to select effective prompt templates for a larger model.
Outcome:	The proposed method significantly reduces the cost of prompt engineering while matching performance with optimal prompts among candidates.

Copied to clipboard

Elke Vandermeerschen, Miryam De Lhoneux

Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study (D18-1)

Challenge:	Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels.
Approach:	They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM.
Outcome:	The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding.

Copied to clipboard

John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, Hong Yu

Challenge:	Existing methods to evaluate deep learning models that are not considered for test set accuracy are difficult to interpret.
Approach:	They examine the impact of a test set question’s difficulty to determine if there is a relationship between difficulty and performance.
Outcome:	The proposed model can learn examples of varying difficulty at different rates if it does well on hard examples and poor on easy items because a dataset is all easy, but has "solved" anything?

SciNLI: A Corpus for Natural Language Inference on Scientific Text (2022.acl-long)

Copied to clipboard

Mobashir Sadat, Cornelia Caragea

What can we learn from Semantic Tagging? (D18-1)

Challenge:	Existing Natural Language Inference (NLI) datasets are not related to scientific text.
Approach:	They propose a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics.
Outcome:	The proposed model achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%.

Copied to clipboard

Mostafa Abdou, Artur Kulmizev, Vinit Ravishankar, Lasha Abzianidze, Johan Bos

Syntactic Text Processing

Challenge:	a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks.
Approach:	They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share .
Outcome:	The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference.

When data permutations are pathological: the case of neural natural language inference (D18-1)

Copied to clipboard

Natalie Schluter, Daniel Varab

Uncertainty Quantification of Large Language Models through Multiple Uncertainty Sources (2026.findings-acl)

Challenge:	Recent research indicates that these datasets are trivially annotated to a large degree.
Approach:	They propose to use a trending NLP task to test a neural architecture for NLI . they show a pathological sensitivity of the systems to permutations of the training set .
Outcome:	The proposed model outperforms the baseline model by multiple percentage points . the proposed model is based on the proposed dataset and the soundness of the task .

Copied to clipboard

Tiejin Chen, Xiaoou Liu, Longchao Da, Jia Chen, Evangelos E. Papalexakis, Hua Wei

Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation (2021.emnlp-main)

Challenge:	Existing methods for uncertainty quantification fail to capture multifaceted nature of natural language generation.
Approach:	They propose a multi-resource Uncertainty Quantification framework that integrates heterogeneous uncertainty signals into a unified measure.
Outcome:	The proposed framework outperforms existing methods on CoQA, NQ_Open, and HotpotQA.

Copied to clipboard

Liyan Xu, Xuchao Zhang, Xujiang Zhao, Haifeng Chen, Feng Chen, Jinho D. Choi

Cross-Lingual Transfer

Challenge:	Recent pre-trained language models have achieved remarkable zero-shot performance . we propose a self-learning framework that utilizes unlabeled data of target languages .
Approach:	They propose a self-learning framework that utilizes unlabeled data of target languages to select silver labels for cross-lingual transfer tasks.
Outcome:	The proposed framework outperforms baseline models on two cross-lingual tasks by 10 F1 on average and 2.5 accuracy on natural language inference (NLI).

Assessing Word Importance Using Models Trained for Semantic Tasks (2023.findings-acl)

Copied to clipboard

Dávid Javorský, Ondřej Bojar, François Yvon

Challenge:	Many NLP tasks require to automatically identify the most significant words in a text.
Approach:	They propose to use attribution methods to explain the predictions of two NLP tasks to derive word significance from models trained to solve semantic tasks.
Outcome:	The proposed method is robust to the initial task and is able to identify important words in sentences without explicit word importance labeling in training.

PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023.findings-acl)

Copied to clipboard

Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Dan Roth, Tal Schuster

Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels (2024.lrec-main)

Challenge:	Existing systems for Natural Language Inference (NLI) only recognize textual entailment relations on sentence-level . however, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence .
Approach:	They propose a system to recognize whether one text is textually entailed by another . they use a corpus of over 45K propositions annotated by human raters to study the textual entailment relation of each proposition in a sentence individually.
Outcome:	The proposed dataset can be used to understand the compositionality of NLI labels.

Copied to clipboard

Panatchakorn Anantaprayoon, Masahiro Kaneko, Naoaki Okazaki

Evaluating Generative Language Models in Information Extraction as Subjective Question Correction (2024.lrec-main)

Challenge:	Existing methods to evaluate gender bias in PLMs focus on one label out of three labels, such as neutral.
Approach:	They propose a bias evaluation method for PLMs that considers all the three labels of NLI task and then defines a measure based on the corresponding label output.
Outcome:	The proposed method can distinguish biased, incorrect inferences from non-biased incorrect infertility better than baseline, resulting in a more accurate bias evaluation.

Copied to clipboard

Yuchen Fan, Yantao Liu, Zijun Yao, Jifan Yu, Lei Hou, Juanzi Li

Atomic Inference for NLI with Generated Facts as Atoms (2024.emnlp-main)

Challenge:	Modern large language models (LLMs) perform poorly in elementary tasks like relation extraction and event extraction due to two issues in conventional evaluation methods.
Approach:	They propose a method to evaluate large language models by incorporating a human annotation schema.
Outcome:	The proposed evaluation method improves matching between model outputs and golden labels.

Copied to clipboard

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Oana-Maria Camburu, Marek Rei

UnNatural Language Inference (2021.acl-long)

Challenge:	Existing models that can provide accurate explanations are not interpretable, i.e. they do not reflect the inner workings of the model.
Approach:	They propose to use LLM-generated facts as atoms to make interpretable models that can be used to make accurate predictions for each component part of an input.
Outcome:	The proposed method outperforms existing methods on natural language understanding tasks with a multi-stage fact generation process and a training regime that incorporates the facts.

Copied to clipboard

Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, Adina Williams

Knowledge-Enhanced Natural Language Inference Based on Knowledge Graphs (2020.coling-main)

Challenge:	Recent studies show that pre-trained NLU models understand human-like syntax . however, these models are word order invariant, causing them to assign gold labels to permutations .
Approach:	They propose to measure the severity of this issue by examining the properties of particular permutations that lead models to be word order invariant.
Outcome:	The proposed model is word order invariant, but it's not human-like syntax.

Copied to clipboard

Zikang Wang, Linjing Li, Daniel Zeng

Knowledge Representation

Challenge:	Existing approaches to natural language inference rely on semantic knowledge, but background knowledge is limited to a few specific types.
Approach:	They propose a Knowledge Graph-enhanced NLI model that leverages background knowledge stored in knowledge graphs to facilitate inference.
Outcome:	The proposed model can leverage background knowledge stored in knowledge graphs to perform the task.

SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data (2025.naacl-long)

Copied to clipboard

Suyoung Bae, YunSeok Choi, Hyojun Kim, Jee-Hyong Lee

Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization (2022.lrec-1)

Challenge:	In many natural language processing tasks, model training often leads to spurious correlations . shortcuts allow models to rely on irrelevant patterns in the data, leading to biased predictions.
Approach:	They propose a method to generate structure-aware positive and negative sentences using tagging.
Outcome:	The proposed method improves model robustness and generalization across different environments while minimizing spurious correlations.

Copied to clipboard

Ramya Akula, Ivan Garibay

The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions (2020.emnlp-main)

Challenge:	Existing evaluation metrics, such as ROUGE and BLEU, rely on exact word matching and fail to capture semantic similarity.
Approach:	They propose to use contextualized word or sentence embeddings to capture semantic similarity between sentences to evaluate text summarization methods.
Outcome:	The proposed evaluation metric shows that it performs faster than the current state-of-the-art on the SummEval dataset.

Copied to clipboard

Xiang Zhou, Yixin Nie, Hao Tan, Mohit Bansal

Challenge:	Neural network models have significantly pushed forward performance on natural language processing benchmarks with the development of largescale language model pre-training.
Approach:	They find that models on natural language inference and reading comprehension are unstable . they propose to use a model-selection routine to analyze the model's instability .
Outcome:	The proposed models can perform poorly on two language-related tasks, the authors show . they also show that the model selection routine is unstable, and that it is not reliable .

Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020.emnlp-main)

Copied to clipboard

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, Tim Rocktäschel

GreekBART: The First Pretrained Greek Sequence-to-Sequence Model (2024.lrec-main)

Challenge:	Neural models pick up on annotation artefacts and spurious correlations, resulting in learning sentences that suffer from the same biases.
Approach:	They propose to tackle this problem by using adversarial training to reduce the bias in sentence representations by using an ensemble of adversaries.
Outcome:	The proposed approach produces more robust models outperforming previous de-biasing efforts when generalised to 12 other NLI datasets.

Copied to clipboard

Iakovos Evdaimon, Hadi Abdine, Christos Xypolopoulos, Stamatis Outsios, Michalis Vazirgiannis, Giorgos Stamou

Neutralizing Bias in LLM Reasoning using Entailment Graphs (2025.findings-acl)

Challenge:	Transfer learning has revolutionized the fields of Computer Vision and Natural Language Processing.
Approach:	They introduce a new language model, GreekBART, that is based on a BART-base architecture.
Outcome:	The proposed model outperforms BERT, GPT and other transformer-based models on discriminative tasks.

Copied to clipboard

Liang Cheng, Tianyi Li, Zhaowei Wang, Tianyang Liu, Mark Steedman

Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)

Challenge:	Natural Language Inference (NLI) is a foundational understanding task in language understanding.
Approach:	They propose a framework to construct counterfactual reasoning data and fine-tune LLMs to reduce attestation bias.
Outcome:	The proposed framework reduces hallucinations from attestation bias on original and bias-neutralized datasets while keeping hypotheses unchanged.

Copied to clipboard

Artur Kulmizev, Erika Lombart, Patrick Watrin, Marie-Catherine de Marneffe

Towards Robustifying NLI Models Against Lexical Dataset Biases (2020.acl-main)

Challenge:	Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks.
Approach:	They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task.
Outcome:	The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns.

Copied to clipboard

Xiang Zhou, Mohit Bansal

Uncertain Natural Language Inference (2020.acl-main)

Challenge:	Recent studies show that deep learning models exploit dataset biases without deep understanding of the language semantics.
Approach:	They propose two methods to debiase models against lexical dataset biases . they use contradiction-word bias and word-overlapping bias as examples .
Outcome:	The proposed method removes label bias at embedding level, while the other uses a bag-of-words sub-model to capture features likely to exploit the bias.

Copied to clipboard

Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, Benjamin Van Durme

Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation (2024.lrec-main)

Challenge:	Existing categorical bin assignments for NLI are not effective for entailment inference.
Approach:	They propose a refinement of Natural Language Inference that targets direct prediction of subjective probability assessments.
Outcome:	The proposed model can be used in pre-training and correlates well with humans.

Copied to clipboard

Mateusz Klimaszewski, Piotr Andruszkiewicz, Alexandra Birch

“Seeing the Big through the Small”: Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? (2024.findings-emnlp)

Challenge:	Existing approaches to modularity are limited to the case of pre-trained modules in a pre-training language model.
Approach:	They propose a method that allows the transfer of pre-trained PEFT modules between incompatible PLMs without any change in the inference complexity.
Outcome:	The proposed method allows the transfer of modules between incompatible PLMs without any change in the inference complexity.

Copied to clipboard

Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, Barbara Plank

Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy (2024.findings-emnlp)

Challenge:	Human label variation arises when multiple human annotators provide different labels for valid reasons.
Approach:	They propose to use crowd workers to represent human judgment distributions or expert linguists to provide detailed explanations for their chosen labels.
Outcome:	The proposed model can approximate human judgment distributions using a small number of expert labels and explanations.

Copied to clipboard

Liyan Xu, Zhenlin Su, Mo Yu, Jin Xu, Jinho Choi, Jie Zhou, Fei Liu

Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations (2025.acl-long)

Challenge:	Existing studies have focused on specialized BERT-variants and recent LLMs to reason inconsistencies.
Approach:	They propose to incorporate task-specific taxonomy into inferences to facilitate both zero-shot and supervised paradigms.
Outcome:	The proposed model outperforms specialized non-LLM and recent LLM models in a number of domains.

Copied to clipboard

Xin Quan, Marco Valentino, Louise A. Dennis, Andre Freitas

CHAmbi: A New Benchmark on Chinese Ambiguity Challenges for Large Language Models (2024.findings-emnlp)

Challenge:	Recent work has shown that the interaction of large language models (LLMs) with theorem provers (TPs) can help verify and improve the validity of NLI explanations.
Approach:	They propose to use logical expressions to guide LLMs in generating structured proof sketches and to use them to improve their accuracy.
Outcome:	The proposed strategies improve autoformalisation, syntactic errors and explanation refinement over the state-of-the-art model.

Copied to clipboard

Qin Zhang, Sihan Cai, Jiaxu Zhao, Mykola Pechenizkiy, Meng Fang

AXCEL: Automated eXplainable Consistency Evaluation using LLMs (2024.findings-emnlp)

Challenge:	Ambiguity is an inherent feature of language, whose management is crucial for effective communication and collaboration.
Approach:	They propose a dataset to evaluate LLMs' ability to handle ambiguity in Chinese by using a specialized Chinese multi-label disambiguation dataset formatted in Natural Language Inference.
Outcome:	The CHAmbi dataset comprises 4,991 pairs of premises and hypotheses, including 824 examples featuring a wide range of ambiguities.

Copied to clipboard

P Aditya Sreekar, Sahil Verma, Suransh Chopra, Abhishek Persad, Sarik Ghazarian, Narayanan Sadagopan

LogicAttack: Adversarial Attacks for Evaluating Logical Consistency of Natural Language Inference (2023.findings-emnlp)

Challenge:	Large Language Models (LLMs) are widely used for various tasks but evaluating the consistency of generated text remains a challenge.
Approach:	They propose a prompt-based consistency metric which provides explanations for consistency scores by providing detailed reasoning and pinpointing inconsistent text spans.
Outcome:	The proposed metric outperforms state-of-the-art metrics in summarization, free text generation and data-to-text conversion tasks by 8.7% and 6.2%.

Copied to clipboard

Mutsumi Nakamura, Santosh Mashetty, Mihir Parmar, Neeraj Varshney, Chitta Baral

Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding (2025.findings-acl)

Challenge:	Recent advances in Large Language Models (LLMs) have demonstrated impressive performance on Natural Language Inference (NLI) tasks.
Approach:	They propose a method to attack NLI models using diverse logical forms of premise and hypothesis using propositional logic to generate effective adversarial attacks.
Outcome:	The proposed method achieves an average 53% Attack Success Rate (ASR) across multiple logic-based attacks.

Copied to clipboard

Elisa Sanchez-Bayona, Rodrigo Agerri

NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports (2023.emnlp-main)

Challenge:	Existing studies on metaphor processing have focused on single datasets and specific task settings, often using artificially constructed data through lexical replacement.
Approach:	They propose to evaluate the capabilities of Large Language Models (LLMs) in metaphor interpretation across multiple datasets, tasks, and prompt configurations.
Outcome:	The proposed frameworks are more realistic and efficient than current models and are more efficient than existing models.

Copied to clipboard

Mael Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, Dónal Landers, Andre Freitas

Rank-Awareness and Angular Constraints: A New Perspective on Learning Sentence Embeddings from NLI Data (2025.emnlp-main)

Challenge:	Clinical trial reports (CTRs) are indispensable for the development of personalized medicine.
Approach:	They propose a resource to help researchers interpret clinical trial reports . they use natural language inference to compute textual entailment .
Outcome:	The proposed resource is the first to cover interpretation of full clinical trial reports . it includes tasks to determine inference relation between natural language statements and CTRs .

Copied to clipboard

Zicheng Zhou, Min Huang, Qinghai Miao

Transitive self-consistency evaluation of NLI models without gold labels (2025.emnlp-main)

Challenge:	High-quality sentence embeddings are critical for advancing a wide range of Natural Language Processing tasks.
Approach:	They propose a framework that leverages the full NLI dataset augmented with pre-computed continuous similarity scores (S) they employ a Rank Margin objective that enforces rank consistency against S using an explicit margin and a Gated Angular objective that conditionally refines embedding geometry based on NLI label (L) and S score agreement.
Outcome:	The proposed framework outperforms baseline models on STS and the MTEB benchmarks.

Copied to clipboard

Wei Wu, Mark Last

Inference Helps PLMs’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs (2024.emnlp-main)

Challenge:	Recent studies show that NLI models make inconsistent model decisions during inference . previous studies tested only variations of two transitive consistency rules out of four .
Approach:	They propose to generate adversarial examples using antonym replacements to test transitive consistency of NLI models.
Outcome:	The proposed approach can test all four rules automatically by generating adversarial examples via antonym replacements.

Copied to clipboard

Juncai Li, Ru Li, Xiaoli Li, Qinghua Chai, Jeff Pan

Reference-Free Evaluation of Taxonomies (2026.findings-acl)

Challenge:	Existing approaches to abstract inference ignore the polysemy and hierarchical nature of concepts . prevailing approaches disregard how arguments might entail differently across various concept levels, thereby missing potential enlargement connections.
Approach:	They propose a framework that organizes arguments hierarchically and delves into entailment relations at diverse concept levels.
Outcome:	The proposed framework improves the model's generalization and reasoning prowess in natural language inference.

Copied to clipboard

Pascal Wullschleger, Majid Zarharan, Donnacha Daly, Marc Pouly, Jennifer Foster

Challenge:	Taxonomies are used to classify items, ideas or organisms based on shared characteristics.
Approach:	They introduce two reference-free metrics for quality evaluation of taxonomies in the absence of labels.
Outcome:	The proposed metrics correlate well with F1 against ground truth taxonomies on five taxonomies and improve hierarchical classification when used with label hierarchies.

SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives (2024.lrec-main)

Copied to clipboard

Rashid Nizamani, Sebastian Schuster, Vera Demberg

Challenge:	scalar implicatures are a phenomenon by which a speaker conveys the negation of a more informative utterance by producing a less informative .
Approach:	They propose to use a dataset to investigate the ability of language models to interpret utterances with scalar implicatures.
Outcome:	The proposed models perform significantly worse on in-domain and out-of-domain examples than other types of NLI examples.

Rule Discovery for Natural Language Inference Data Generation Using Out-of-Distribution Detection (2025.emnlp-main)

Copied to clipboard

Juyoung Han, Hyunsun Hwang, Changki Lee

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations (2026.findings-acl)

Challenge:	Existing training rules for natural language inference do not cover the diversity of natural language.
Approach:	They propose a framework that combines out-of-distribution detection and clustering to identify new premise–hypothesis pairs in a dataset that are not covered by existing rules.
Outcome:	The proposed framework achieves +0.85%p accuracy on 2k and +0.15%p on 550k samples.

Copied to clipboard

Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Benjamin Roth, Barbara Plank

Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues (2024.lrec-main)

Challenge:	Natural Language Inference (NLI) datasets often exhibit label variation.
Approach:	They extend LiTEx taxonomy to two NLI datasets and jointly analyze label variation and label variation.
Outcome:	The proposed model combines explanations as a lens to analyze variation in NLI annotations and examine individual differences in reasoning.

Copied to clipboard

Maria Francis, Julius Steuer, Dietrich Klakow, Volha Petukhova

Information Retrieval

Challenge:	Existing models for presupposition generation fail to generate complete lists of presuffpositions.
Approach:	They propose to fine-tune existing BERT and T5 models for a task where a model produces a list of presuppositions carried by the given input sentence.
Outcome:	The proposed models outperform BERT and T5 models on the novel task of presupposition as natural language inference (PNLI) despite limited data, they achieved an emerging proficiency in generation of presumptions reaching ROUGE scores of 43.47, adhering to systematic patterns that mirror valid strategies for pres upposition generation, although failed to generate the complete lists.

Entailed Between the Lines: Incorporating Implication into NLI (2025.acl-long)

Copied to clipboard

Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya, Alex Fabrikant

CiteEval: Principle-Driven Citation Evaluation for Source Attribution (2025.acl-long)

Challenge:	True Emotions, social cues, insults, and a myriad of other messages are conveyed implicitly, often even more so than explicitly.
Approach:	They propose a dataset to help LLMs understand implied entailment .
Outcome:	The proposed dataset enables LLMs to understand implied entailment and can generalize this understanding across datasets and domains.

Copied to clipboard

Yumo Xu, Peng Qi, Jifan Chen, Kunlun Liu, Rujun Han, Lan Liu, Bonan Min, Vittorio Castelli, Arshit Gupta, Zhiguo Wang

Challenge:	Current evaluation frameworks rely on NLI to assess binary or ternary support from cited sources, which is suboptimal for citation evaluation.
Approach:	They propose a citation evaluation framework based on fine-grained citation ratings within a broad context and construct a multi-domain benchmark with high-quality human annotations.
Outcome:	The proposed framework provides a high-quality human annotation benchmark and a suite of model-based metrics that exhibit strong correlation with human judgments.

Cross-Document Cross-Lingual NLI via RST-Enhanced Graph Fusion and Interpretability Prediction (2025.emnlp-main)

Copied to clipboard

Mengying Yuan, WenHao Wang, Zixuan Wang, Yujie Huang, Kangli Wei, Fei Li, Chong Teng, Donghong Ji

Extractive Fact Decomposition for Interpretable Natural Language Inference in one Forward Pass (2025.emnlp-main)

Challenge:	Despite the development of many subdirections, Cross-Document Cross-Lingual NLI remains largely unexplored.
Approach:	They propose a novel paradigm that extends traditional NLI capabilities to multi-document, multilingual scenarios by integrating RST-enhanced graph fusion with interpretability-aware prediction.
Outcome:	The proposed method improves on existing models and document-level NLI to multi-document, multilingual scenarios.

Copied to clipboard

Nicholas Popovič, Michael Färber

LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference (2025.emnlp-main)

Challenge:	Recent work in Natural Language Inference (NLI) uses atomic fact decomposition to enhance interpretability and robustness.
Approach:	They propose an encoder-only architecture that performs extractive atomic fact decomposition and interpretable inference without generative models.
Outcome:	The proposed architecture achieves competitive accuracy and improves robustness out of distribution and in adversarial settings over models based on extractive rationale supervision.

Copied to clipboard

Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Barbara Plank

ViLegalLM: Language Models for Vietnamese Legal Text (2026.findings-acl)

Challenge:	Existing evidence of human label variation in Natural Language Inference (NLI) however, within-label variation is an additional challenge.
Approach:	They propose a linguistically-informed taxonomy for categorizing free-text explanations in English that captures different reasoning strategies behind NLI explanations with a particular focus on within-label variation.
Outcome:	The proposed taxonomy can be used to classify explanations in English using a linguistically-informed taxonomies.

Copied to clipboard

Truong-Phuc Nguyen, Quy-Nhan Nguyen, Minh-Tien Nguyen